Image processing device

ABSTRACT

An image processing device according to an embodiment exemplarily includes, as an example: a processor that acquires a taken image obtained by imaging surroundings of a vehicle by an imaging unit mounted on the vehicle; projects the taken image to a stereoscopic projection plane including the vehicle and a first projection region rising above the vehicle and produces a composite image of the stereoscopic projection plane when viewed from a virtual viewpoint; causes a display to display the composite image; determines a target parking region in which the vehicle is parked in the stereoscopic projection plane; and shifts at least a part of a movement path of the vehicle from a position of the vehicle to the target parking region and the target parking region of the vehicle to a second projection region horizontal to the vehicle.

TECHNICAL FIELD

The present invention relates to an image processing device.

BACKGROUND ART

Techniques have been developed that project a taken image obtained byimaging surroundings of a vehicle on a three-dimensional projectionplane and display a composite image of the three-dimensional projectionplane to which the taken image is projected when viewed from a virtualviewpoint.

CITATION LIST Patent Literature

Patent Document 1: Japanese Patent No. 5292874

Patent Document 2: Japanese Patent No. 3300334

SUMMARY OF INVENTION Problem to be Solved by the Invention

When a flat projection plane including a vehicle in thethree-dimensional projection plane is narrow, a parking region presentaround the vehicle is not projected to the flat projection plane,thereby causing a parking frame that makes it possible to identify theparking region to be distorted. On the other hand, when the flatprojection plane is too wide, an obstacle at a position far from thevehicle is projected to the flat projection plane, thereby causing theobstacle to be distorted. As a result, the composite image differentfrom a natural image is displayed.

Means for Solving Problem

An image processing device of an embodiment includes, for example: anacquisition unit that acquires a taken image obtained by imagingsurroundings of a vehicle by an imaging unit mounted on the vehicle; ageneration unit that projects the taken image to a stereoscopicprojection plane including the vehicle and a first projection regionrising above the vehicle and, produces a composite image of thestereoscopic projection plane when viewed from a virtual viewpoint; adisplay control nit that causes a display to display the compositeimage; a decision unit that determines a target parking region in whichthe vehicle is parked in the stereoscopic projection plane; and a changeunit that shifts at least a part of a movement path of the vehicle froma position of the vehicle to the target parking region and the targetparking region of the vehicle to a second projection region horizontalto the vehicle. The image processing device in the embodiment, thus, candisplay the composite image in which the target parking region of thevehicle is visually recognized naturally, for example.

In the image processing device of the embodiments, wherein, the firstprojection region is positioned around the second projection region, thesecond projection region includes the vehicle, and the change unitenlarges the second projection region as a whole such that an end on afar side of the target parking region when viewed from the position ofthe vehicle is included in the second projection region. The imageprocessing device in the embodiment, thus, can display the compositeimage in which the target parking region of the vehicle is visuallyrecognized naturally, for example.

In the image processing device of the embodiments, wherein the changeunit partially enlarges the second projection region such that only atleast a part of the target parking region and the movement path in thefirst projection region is included in the second projection region. Theimage processing device in the embodiment, thus, can display thecomposite image in which the target parking region of the vehicle and anobstacle around the target parking region are visually recognizednaturally, for example.

In the image processing device of the embodiments, wherein, the firstprojection region is positioned around the second projection region, thesecond projection region includes the vehicle, and the change unitenlarges the second projection region such that an end on a far side ofthe target parking region when viewed from the position of the vehicleis included in the second projection region when an obstacle is absentaround the target parking region, and partially enlarges the secondprojection region such that only a part of the target parking region andthe movement path in the first projection region is included in thesecond projection region when an obstacle is present around the targetparking region. The image processing device in the embodiment, thus, candisplay the composite image in which the target parking region and thesurroundings of the target parking region are visually recognizednaturally, even when the target parking position and a surroundingenvironment of the vehicle are changed, for example.

In the image processing device of the embodiments, further comprising: astorage that stores therein a first map associating coordinates in thestereoscopic projection plane with texture coordinates of the takenimage projected to the coordinates and a second map associatingcoordinates in a planar surface projection plane, the planar surfaceprojection plane being composed of a single projection plane horizontalto the vehicle, with the texture coordinates projected to thecoordinates; and a transmission unit that, when the target parkingregion is decided, overwrites a map associating coordinates of themovement path in the second projection region with the texturecoordinates of the taken image projected to the coordinates, the mapbeing included in the first map, on the second map associatingcoordinates of the movement path in the planar surface projection planewith the texture coordinates of the taken image projected to thecoordinates, and transmits the first map to the generation unit, whereinthe generation unit projects the taken image to the stereoscopicprojection plane in accordance with the first map received from thetransmission unit. The image processing device in the embodiment, thus,can accelerate display processing of the composite image.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a perspective view illustrating an exemplary state where apart of a cabin of a vehicle on which an image processing deviceaccording to a first embodiment is mounted is viewed in a perspectivemanner;

FIG. 2 is an exemplary plan view of the vehicle according to the firstembodiment;

FIG. 3 is a block diagram illustrating an exemplary functional structureof the vehicle according to the first embodiment;

FIG. 4 is a block diagram illustrating an exemplary functional structureof an ECU included in the vehicle according to the first embodiment;

FIG. 5 is a flowchart illustrating an exemplary flow of displayprocessing of the composite image by the ECU included in the vehicleaccording to the first embodiment;

FIG. 6 is a diagram for explaining exemplary change processing of astereoscopic projection region by the ECU included in the vehicleaccording to the first embodiment;

FIG. 7 is a diagram for explaining other exemplary change processing ofthe stereoscopic projection region by the ECU included in the vehicleaccording to the first embodiment;

FIG. 8 is a diagram illustrating an exemplary composite image displayedby the ECU included in the vehicle according to the first embodiment;

FIG. 9 is an exemplary cross-sectional view of a stereoscopic projectionplane to which a taken image is projected by the ECU included in thevehicle according to the first embodiment;

FIG. 10 is a diagram illustrating another exemplary composite imagedisplayed by the ECU included in the vehicle according to the firstembodiment;

FIG. 11 is another exemplary cross-sectional view of the stereoscopicprojection plane to which a taken image is projected by the ECU includedin the vehicle according to the first embodiment;

FIG. 12 is a diagram illustrating another exemplary composite imagedisplayed by the ECU included in the vehicle according to the firstembodiment:

FIG. 13 is another exemplary cross-sectional view of the stereoscopicprojection plane to which a taken image is projected by the ECU includedin the vehicle according to the first embodiment;

FIG. 14 is a diagram illustrating another exemplary composite imagedisplayed by the ECU included in the vehicle according to the firstembodiment;

FIG. 15 is another exemplary cross-sectional view of the stereoscopicprojection plane to which a taken image is projected by the ECU includedin the vehicle according to the first embodiment;

FIG. 16 is a diagram illustrating another exemplary composite imagedisplayed by the ECU included in the vehicle according to the firstembodiment;

FIG. 17 is another exemplary cross-sectional view of the stereoscopicprojection plane to which a taken image is projected by the ECU includedin the vehicle according to the first embodiment;

FIG. 18 is a diagram illustrating another exemplary composite imagedisplayed by the ECU included in the vehicle according to the firstembodiment;

FIG. 19 is another exemplary cross-sectional view of the stereoscopicprojection plane to which a taken image is projected by the ECU includedin the vehicle according to the first embodiment; and

FIG. 20 is a diagram for explaining exemplary projection processing ofthe taken image by the ECU included in the vehicle according to a secondembodiment.

DESCRIPTION OF EMBODIMENTS

The following discloses exemplary embodiments of the invention. Thestructures of the following embodiments and operations, results, andeffects brought by the structures are examples. The invention can beachieved by structures other than those disclosed by the followingembodiments and the structures can obtain at least one of variouseffects based on the basic structures and derivative effects.

The vehicle on which an image processing device according to theembodiments is mounted may be an automobile (internal combustion engineautomobile) having an internal combustion engine as its drive source, anautomobile (electric automobile, fuel battery automobile, or the like)having an electric motor (motor) as its drive source, or an automobile(hybrid automobile) having both the internal combustion engine and theelectric motor as its drive source. On the vehicle, various speedchanging devices and various devices (systems, parts, and the like)necessary for driving the internal combustion engine and the electronicmotor can also be mounted. The systems, the number, and the layout ofdevices for driving wheels can be set in various manners in the vehicle.

First Embodiment

FIG. 1 is a perspective view illustrating an exemplary state where apart of a cabin of a vehicle on which an image processing deviceaccording to a first embodiment is mounted is viewed in a perspectivemanner. As illustrated in FIG. 1, a vehicle 1 includes a vehicle body 2,a steering unit 4, an accelerating operation unit 5, a braking operationunit 6, a speed shifting operation unit 7, and a monitoring device 11.The vehicle body 2 has a vehicle cabin 2 a in which occupants sit. Inthe vehicle cabin 2 a, the steering unit 4, the accelerating operationunit 5, the braking operation unit 6, and the speed shifting operationunit 7, for example, are provided such that a driver who is the occupantand sits on a seat 2 b faces them. The steering unit 4 is a steeringwheel protruding from a dashboard 24, for example. The acceleratingoperation unit 5 is an accelerator pedal positioned near the place wherethe driver's foot naturally rests, for example. The braking operationunit 6 is a brake pedal positioned near the place where the driver'sfoot naturally rests, for example. The speed shifting operation unit 7is a shift lever protruding from a center console, for example.

The monitoring device 11 is provided to the central portion in a vehiclewidth direction (i.e., left-right direction) of the dashboard 24, forexample. The monitoring device 11 may have a function of a navigationsystem or an audio system, for example. The monitoring device 11 has adisplay 8, a voice output device 9, and an operation input unit 10. Themonitoring device 11 may have various operation input units such as aswitch, a dial, a joystick, and a push button.

The display 8 includes a liquid crystal display (LCD) or an organicelectroluminescent display (OELD), for example, and can display variousimages on the basis of image data. The voice output device 9 includes aspeaker, for example, and outputs various voices on the basis of voicedata. The voice output device 9 may be provided at a position differentfrom that of the monitoring device 11 in the vehicle cabin 2 a.

The operation input unit 10 includes a touch panel, for example, andallows the occupant to input various types of information to it. Theoperation input unit 10 is provided on a display screen of the display 8and allows the image displayed on the display 8 to pass through theoperation input unit 10. The operation input unit 10, thus, allows theoccupant to visually recognize the image displayed on the display screenof the display 8. The operation input unit 10 detects the occupant'stouch operation on the display screen of the display 8, therebyreceiving various types of information input by the occupant.

FIG. 2 is an exemplary plan view of the vehicle according to the firstembodiment. As illustrated in FIGS. 1 and 2, the vehicle 1 is afour-wheel automobile, for example, and has two front wheels 3F on theleft and the right sides and two rear wheels 3R on the left and theright sides. All or a part of the four wheels 3 can be steered.

The vehicle 1 includes a plurality of imaging units 15 mounted thereon.In the embodiment, four imaging units 15 a to 15 d are mounted on thevehicle 1, for example. The imaging unit 15 is a digital camera havingan imaging element such as a charge coupled device (CCD) or a CMOS imagesensor (CIS). The imaging unit 15 can image surroundings of the vehicle1 at a certain frame rate. The imaging unit 15 outputs the taken imageobtained by imaging the surroundings of the vehicle 1. The imaging unit15 has a wide-angle lens or a fish-eye lens and can image a range from140 degrees to 220 degrees in the horizontal direction, for example.Optical axes of the imaging units 15 are set obliquely downward in somecases.

Specifically, the imaging unit 15 a is positioned at an end portion 2 eon the rear side of the vehicle body 2 and provided on the wall portionon the lower side of a rear window of a door 2 h of a rear hatch, forexample. The imaging unit 15 a can image a rear region of the vehicle 1in the surroundings of the vehicle 1. The imaging unit 15 b ispositioned at an end portion 2 f on the right side of the vehicle body 2and provided to a door mirror 2 g on the right side, for example. Theimaging unit 15 b can image a side region of the vehicle 1 in thesurroundings of the vehicle 1. The imaging unit 15 c is positioned at anend portion 2 c on the front side of the vehicle body 2, i.e., on thefront direction side in the front-rear direction of the vehicle 1, andprovided to a front bumper or a front grill, for example. The imagingunit 15 c can image a front region of the vehicle 1 in the surroundingsof the vehicle 1. The imaging unit 15 d is positioned on the left sideof the vehicle body 2, i.e., at an end portion 2 d on the left side inthe vehicle width direction, and provided to a door mirror 2 g on theleft side. The imaging unit 15 d can image a side region of the vehicle1 in the surroundings of the vehicle 1.

FIG. 3 is a block diagram illustrating an exemplary functional structureof the vehicle according to the first embodiment. As illustrated in FIG.3, the vehicle 1 includes a steering system 13, a braking system 18, asteering angle sensor 19, an accelerator sensor 20, a shift sensor 21, awheel speed sensor 22, an in-vehicle network 23, and an electroniccontrol unit (ECU) 14. The monitoring device 11, the steering system 13,the braking system 18, the steering angle sensor 19, the acceleratorsensor 20, the shift sensor 21, the wheel speed sensor 22, and the ECU14 are electrically coupled with the in-vehicle network 23 serving as anelectrical communication line. The in-vehicle network 23 is a controllerarea network (CAN), for example.

The steering system 13 is an electric power steering system or a steerby wire (SBW) system, for example. The steering system 13 has anactuator 13 a and a torque sensor 13 b. The steering system 13, which iselectrically controlled by the ECU 14, for example, applies torque tothe steering unit 4 so as to add steering force to the steering unit 4by operating the actuator 13 a, thereby steering the wheels 3. Thetorque sensor 13 b detects torque applied to the steering unit 4 by thedriver and transmits the detection result to the ECU 14.

The braking system 18 includes an anti-lock brake system (ABS) thatcontrols locking up during braking in the vehicle 1, an electronicstability control (ESC) that prevents skidding of the vehicle 1 incornering, an electric braking system that increases braking force toassist the brake operation, and a brake by wire (BBW). The brakingsystem 18 has an actuator 18 a and a brake sensor 18 b. The brakingsystem 18, which is electrically controlled by the ECU 14, for example,applies braking force to the wheels 3 via the actuator 18 a. The brakingsystem 18 detects indications of locking up of the brake, idling of thewheels 3, and the skidding, for example, from a difference in rotationbetween the wheels 3 on the right side and on the left side, forexample, and performs control so as to prevent the locking up of thebrake, the idling of the wheels 3, and the skidding. The brake sensor 18b is a displacement sensor that detects the position of the brake pedalserving as a movable unit of the braking operation unit 6, and transmitsthe detection result of the position of the brake pedal to the ECU 14.

The steering angle sensor 19 detects a steering amount of the steeringunit 4 such as a steering wheel. In the embodiment, the steering anglesensor 19 includes a hall element, for example, detects a rotation angleof the rotating portion of the steering unit 4 as the steering amount,and transmits the detection result to the ECU 14. The accelerator sensor20 is a displacement sensor that detects the position of the acceleratorpedal serving as the movable unit of the accelerating operation unit 5,and transmits the detection result to the ECU 14.

The shift sensor 21 detects the position of the movable units (e.g., abar, an arm, and a button) of the speed shifting operation unit 7 andtransmits the detection result to the ECU 14. The wheel speed sensor 22has hall elements, for example, detects rotation amounts of the wheels 3and the number of rotations of wheels 3 per unit time, and transmits thedetection result to the ECU 14.

The ECU 14 functions as an example of the image processing device thatprojects the taken image obtained by imaging the surroundings of thevehicle 1 by the imaging unit 15 to a preliminarily set projectionplane, produces an image of the projection plane when viewed from avirtual viewpoint, and causes the display 8 to display the producedimage. The ECU 14 is a computer, for example. The ECU 14 manages thewhole control of the vehicle 1 by cooperation of hardware and software.Specifically, the ECU 14 includes a central processing unit (CPU) 14 a,a read only memory (ROM) 14 b, a random access memory (RAM) 14 c, adisplay control unit 14 d, a voice control unit 14 e, and a solid statedrive (SSD) 14 f. The CPU 14 a, the ROM 14 b, and the RAM 14 c may beprovided in the same circuit substrate.

The CPU 14 a reads out a program stored in a nonvolatile storage devicesuch as the ROM 14 b and executes various types of arithmetic processingin accordance with the program. For example, the CPU 14 a performs imageprocessing on image data that the CPU 14 a causes the display 8 todisplay.

The ROM 14 b stores therein various programs and parameters necessaryfor executing the programs, for example. The RAM 14 c temporarily storestherein various types of data used for arithmetic operation by the CPU14 a. The display control unit 14 d performs mainly the image processingon the image data that is acquired from the imaging unit 15 and outputto the CPU 14 a, and conversion from the image data acquired from theCPU 14 a to the display image data that the imaging unit 15 causes thedisplay 8 to display, out of the arithmetic processing by the ECU 14,for example. The voice control unit 14 e mainly performs processing on avoice that is acquired from the CPU 14 a and the voice control unit 14 ecauses the voice output device 9 to output, out of the arithmeticprocessing by the ECU 14. The SSD 14 f, which is a writable nonvolatilestorage unit, continues to store therein the data acquired from the CPU14 a even when the power source of the ECU 14 is turned off.

FIG. 4 is a block diagram illustrating an exemplary functional structureof the ECU included in the vehicle according to the first embodiment. Asillustrated in FIG. 4, the ECU 14 includes a vehicle state acquisitionunit 401, an image acquisition unit 402, a composite image generationunit 403, a vehicle surrounding situation detection unit 404, a targetparking region decision unit 405, a projection region change unit 406,and an image output unit 407. For example, the processor such as the CPU14 a mounted on the circuit substrate executes a surroundings monitoringprogram stored in the storage medium such as the ROM 14 b or the SSD 14f. As a result, the ECU 14 achieves the functions of the vehicle stateacquisition unit 401, the image acquisition unit 402, the compositeimage generation unit 403, the vehicle surrounding situation detectionunit 404, the target parking region decision unit 405, the projectionregion change unit 406, and the image output unit 407. A part or all ofthe vehicle state acquisition unit 401, the image acquisition unit 402,the composite image generation unit 403, the vehicle surroundingsituation detection unit 404, the target parking region decision unit405, the projection region change unit 406, and the image output unit407 may be structured by hardware such as a circuit.

The vehicle state acquisition unit 401 acquires the state of the vehicle1 such as the steering amount of the steering unit 4 detected by thesteering angle sensor 19, the position of the accelerator pedal detectedby the accelerator sensor 20, the position of the movable unit of thespeed shifting operation unit 7 detected by the shift sensor 21, and thenumber of rotations of the wheels 3 detected by the wheel speed sensor22. The image acquisition unit 402 acquires, from the imaging unit 15,the taken image obtained by imaging the surroundings of the vehicle 1 bythe imaging unit 15.

The composite image generation unit 403 projects the taken imageacquired by the image acquisition unit 402 to a three-dimensionalprojection plane (hereinafter, described as a stereoscopic projectionplane). The stereoscopic projection plane is a projection plane thatincludes the vehicle 1 and a projection region (hereinafter, describedas a stereoscopic projection region) rising above the vehicle 1. In theembodiment, the stereoscopic projection plane has a projection regionhorizontal to the vehicle 1 (hereinafter, described as a planar surfaceprojection region) and the stereoscopic projection region positionedaround the planar surface projection region. The composite imagegeneration unit 403 projects the taken image to the stereoscopicprojection plane and produces an image of the stereoscopic projectionplane to which the taken image is projected when viewed from the virtualviewpoint (hereinafter, the image is described as a composite image).

The vehicle surrounding situation detection unit 404 detects, on thebasis of the taken image acquired by the image acquisition unit 402, forexample, the surrounding situation of the vehicle 1 such as a targetparking position that is the position at which the vehicle 1 is parkedand an obstacle present around the vehicle 1. The target parking regiondecision unit 405 determines a target parking region in which thevehicle 1 is parked in the stereoscopic projection plane. The targetparking region corresponds to the target parking position detected bythe vehicle surrounding situation detection unit 404 in the stereoscopicprojection plane. Specifically, the target parking region is the regionto which the taken image of the target parking position (taken imageobtained by imaging the target parking position of the vehicle 1 by theimaging unit 15) is projected in the stereoscopic projection plane.

The image output unit 407 functions as an example of the display controlunit that outputs the composite image produced by the composite imagegeneration unit 403 to the display 8 to cause the display 8 to displaythe composite image. The projection region change unit 406 shifts atleast a part of a movement path and the target parking region of thevehicle 1 in the stereoscopic projection region to the planar surfaceprojection region. The movement path is a path along which the vehicle 1moves from its position to the target parking region. As a result of theshift, the taken image of the movement path of the vehicle 1 isprojected to a flat projection plane, thereby making it possible toprevent the target parking region (e.g., a parking frame) of the vehicle1 from being distorted in the stereoscopic projection plane or not beingprojected to the stereoscopic projection plane. As a result, thecomposite image can be displayed in which the target parking region ofthe vehicle 1 can be visually recognized naturally.

The following describes an exemplary flow of display processing of thecomposite image by the ECU 14 included in the vehicle 1 according to theembodiment with reference to FIG. 5. FIG. 5 is a flowchart illustratingan exemplary flow of the display processing of the composite image bythe ECU included in the vehicle according to the first embodiment.

In the embodiment, the image acquisition unit 402 acquires, from theoperation input unit 10, for example, a display instruction thatinstructs display of the composite image (step S501). If the displayinstruction is received (Yes at step S502), the image acquisition unit402 acquires, from the imaging unit 15, the taken image obtained byimaging the surroundings of the vehicle 1 by the imaging unit 15 (stepS503). For example, the image acquisition unit 402 acquires, at thecurrent position of the vehicle 1, the taken image obtained by imagingthe surroundings of the vehicle 1 by the imaging unit 15.

The composite image generation unit 403 projects the acquired takenimage to the stereoscopic projection plane and produces the compositeimage of the stereoscopic projection plane when viewed from thepreliminarily set virtual viewpoint (step S504). In the embodiment, thecomposite image generation unit 403 converts respective coordinates(hereinafter, described as texture coordinates) in the acquired takenimage into the coordinates in the stereoscopic projection plane byprojection conversion. The composite image generation unit 403 projectsthe images at the respective texture coordinates in the acquired takenimage to the positions represented by the coordinates converted from thetexture coordinates by the projection conversion. The composite imagegeneration unit 403, thus, projects the taken image to the stereoscopicprojection plane.

The vehicle surrounding situation detection unit 404 detects thesurrounding situation of the vehicle 1 such as the target parkingposition on the basis of the taken image acquired by the imageacquisition unit 402, for example (step S505). The target parking regiondecision unit 405 determines, in the stereoscopic projection plane, thetarget parking region corresponding to the target parking positiondetected by the vehicle surrounding situation detection unit 404 (stepS506). The composite image generation unit 403 determines whether thedetermined target parking region is included in the stereoscopicprojection region (step S507). If the determined target parking regionis not included in the stereoscopic projection region (No at step S507),the image output unit 407 causes the display 8 to display the compositeimage produced by the composite image generation unit 403 (step S508).

If the determined target parking region is included in the stereoscopicprojection region (Yes at step S507), the composite image generationunit 403 shifts the movement path and the target parking region of thevehicle 1 in the stereoscopic projection region to the planar surfaceprojection region (step S509). The composite image generation unit 403again projects the taken image to the stereoscopic projection planeafter the stereoscopic projection region is changed and produces thecomposite image of the stereoscopic projection plane to which the takenimage is again projected when viewed from the virtual viewpoint (stepS510). Thereafter, the image output unit 407 causes the display 8 todisplay the composite image produced by the composite image generationunit 403 (step S508).

The following describes an exemplary change processing of thestereoscopic projection region by the ECU 14 included in the vehicle 1according to the embodiment with reference to FIG. 6. FIG. 6 is adiagram for explaining the exemplary change processing of thestereoscopic projection region by the ECU included in the vehicleaccording to the first embodiment. In FIG. 6, a direction that isparallel to a planar surface projection region R2 is defined as a Zdirection, a direction that is parallel to the planar surface projectionregion R2 and perpendicular to the Z direction is defined as an Xdirection, and a direction that is vertical to the planar surfaceprojection region R2 is defined as a Y direction.

For example, as illustrated in FIG. 6, the composite image generationunit 403 preliminarily produces a stereoscopic projection plane Rincluding the planar surface projection region R2 and a stereoscopicprojection region R1. The planar surface projection region R2 is a flatprojection region having a circular shape (e.g., circle or ellipse) witha position P of the vehicle 1 as a center. In the embodiment, the planarsurface projection region R2 is a circular projection region. The shapeis not limited to this shape. For example, the planar surface projectionregion R2 may be a flat projection region having a polygonal shape. Thestereoscopic projection region R1 is a projection region that graduallyrises from the end (outer edge) of the planar surface projection regionR2 serving as a reference in the Y direction (above the planar surfaceprojection region R2) as it goes away from the planar surface projectionregion R2. Specifically, the stereoscopic projection region R1 is acurved surface that rises from the outer edge of the planar surfaceprojection region R2 in the Y direction in an elliptic shape or aparabolic shape. The composite image generation unit 403 produces, asthe stereoscopic projection plane R, a three-dimensional surface thathas a bowl-like shape and includes the position P of the vehicle 1.Thereafter, the composite image generation unit 403 projects the takenimage acquired by the image acquisition unit 402 to the producedstereoscopic projection plane R.

The vehicle surrounding situation detection unit 404 detects the targetparking position of the vehicle 1 on the basis of the taken imageacquired by the image acquisition unit 402, for example. As illustratedin FIG. 6, the target parking region decision unit 405 determines atarget parking region TR corresponding to the target parking positiondetected by the vehicle surrounding situation detection unit 404 in thestereoscopic projection plane R. As illustrated in FIG. 6, when thedetermined target parking region TR is included in the stereoscopicprojection region R1, the projection region change unit 406 increases aradius of the planar surface projection region R2 such that an end E onthe far side of the target parking region TR when viewed from theposition P of the vehicle 1 in the stereoscopic projection region R1 isincluded in the planar surface projection region R2. In the embodiment,the projection region change unit 406 increases the radius of the planarsurface projection region R2 so as to include the end E on the far sideof the target parking region TR when viewed from the position P of thevehicle 1 in the planar surface projection region R2. The end E on thefar side of the target parking region TR when viewed from the position Pof the vehicle 1 is not limited to being included in the planar surfaceprojection region R2 in the manner described above. The end E isincluded in the planar surface projection region R2 in any manner toincrease (enlarge) the planar surface projection region R2 as a whole.

This change makes it possible to project the taken image of the movementpath and the target parking region TR of the vehicle 1 to the flatplanar surface projection region R2. This, thus, can prevent the image(e.g., the parking frame) that makes it possible to identify the targetparking position of the vehicle 1 from being distorted in thestereoscopic projection plane R or not being projected to thestereoscopic projection plane R. As a result, the composite image can bedisplayed in which the target parking region TR of the vehicle 1 isvisually recognized naturally.

The following describes other exemplary change processing of thestereoscopic projection region by the ECU 14 included in the vehicle 1according to the embodiment with reference to FIG. 7. FIG. 7 is adiagram for explaining other exemplary change processing of thestereoscopic projection region by the ECU included in the vehicleaccording to the first embodiment. In FIG. 7, a direction that isparallel to the planar surface projection region R2 is defined as the Zdirection, a direction that is parallel to the planar surface projectionregion R2 and perpendicular to the Z direction is defined as the Xdirection, and a direction that is vertical to the planar surfaceprojection region R2 is defined as the Y direction.

For example, as illustrated in FIG. 7, the composite image generationunit 403 projects the taken image acquired by the image acquisition unit402 to the preliminarily produced stereoscopic projection plane R in thesame manner as described with reference to FIG. 6. The vehiclesurrounding situation detection unit 404 detects the target parkingposition of the vehicle 1 on the basis of the taken image acquired bythe image acquisition unit 402, for example. As illustrated in FIG. 7,the target parking region decision unit 405 determines the targetparking region TR corresponding to the target parking position detectedby the vehicle surrounding situation detection unit 404 in thestereoscopic projection plane R.

As illustrated in FIG. 7, when the decided target parking region TR isincluded in the stereoscopic projection region R1, the projection regionchange unit 406 partially increases (enlarges) the planar surfaceprojection region R2 such that only the target parking region TR in thestereoscopic projection region R1 is included in the planar surfaceprojection region R2. In other words, as illustrated in FIG. 7, when thedecided target parking region TR is included in the stereoscopicprojection region R1, the composite image generation unit 403 enlargesthe planar surface projection region R2 such that the target parkingregion TR is included in the planar surface projection region R2 whilethe region other than the target parking region TR in the stereoscopicprojection region R1 remains in the stereoscopic projection region R1.

When a three-dimensional object such as another vehicle is present in aregion adjacent to the target parking region TR, the change describedabove makes it possible to project the taken image of thethree-dimensional object that is projected to the region adjacent to thetarget parking region TR to the stereoscopic projection region R1 whilethe taken image of the target parking region TR is projected to the flatplanar surface projection region R2. The image of the obstacle projectedto the stereoscopic projection plane R, thus, does not become anelongated image, thereby making it possible to prevent the targetparking region TR (e.g., parking frame) of the vehicle 1 from beingdistorted. As a result, the composite image can be displayed in whichthe target parking region TR of the vehicle 1 and the obstacle aroundthe target parking region TR are visually recognized naturally. In theembodiment, the projection region change unit 406 shifts the whole ofthe target parking region TR in the stereoscopic projection region R1 tothe planar surface projection region R2. The planar surface projectionregion R2 can be enlarged in any manner satisfying that only at least apart of the target parking region TR and the movement path in thestereoscopic projection region R1 is included in the planar surfaceprojection region R2.

The projection region change unit 406 can switch the methods forchanging the stereoscopic projection region R1 to the planar surfaceprojection region R2 in accordance with whether an obstacle is presentaround the decided target parking region TR. Specifically, when noobstacle is present around the target parking region TR, the projectionregion change unit 406 enlarges the planar surface projection region R2such that the end E on the far side of the target parking region TR whenviewed from the position P of the vehicle 1 is included in the planarsurface projection region R2. When an obstacle is present around thetarget parking region TR, the projection region change unit 406 enlargesthe planar surface projection region R2 such that only at least a partof the target parking region TR and the movement path in thestereoscopic projection region R1 is included in the planar surfaceprojection region R2. As a result, the composite image can be displayedin which the target parking region TR and the surroundings of the targetparking region TR are visually recognized naturally even when anenvironment around the target parking position of the vehicle 1 ischanged.

When a region far from (ahead) the target parking region TR when viewedfrom the vehicle 1 is included in the planar surface projection regionR2 in the stereoscopic projection plane R, the projection region changeunit 406 can reduce the planar surface projection region R2.Specifically, the projection region change unit 406 reduces the planarsurface projection region R2 such that the end E on the far side of thetarget parking region TR when viewed from the vehicle 1 coincides withthe end of the planar surface projection region R2 in the stereoscopicprojection plane R. This reduction makes it possible to project theregion far from the target parking region TR to the stereoscopicprojection region R1 while the taken image of the target parking regionTR is projected to the planar surface projection region R2.

The following describes examples of the composite image displayed by theECU 14 included in the vehicle 1 according to the embodiment withreference to FIGS. 8 to 13. FIGS. 8, 10, and 12 are diagramsillustrating the examples of the composite image displayed by the ECUincluded in the vehicle according to the first embodiment. FIGS. 9, 11,and 13 are exemplary cross-sectional views of the stereoscopicprojection plane to which the taken image is projected by the ECUincluded in the vehicle according to the first embodiment. In FIGS. 9,11, and 13, the abscissa axis represents the distance from the positionof the vehicle 1 to each point in the stereoscopic projection plane onthe XZ plane parallel to the X direction and the Z direction while thevertical axis represents the height (i.e., the coordinate in the Ydirection) of each point in the stereoscopic projection plane.

For example, as illustrated in FIG. 9, the composite image generationunit 403 projects the taken image to the stereoscopic projection planeincluding a circular planar surface projection region having a radius of5000 mm with the position of the vehicle 1 as a center and astereoscopic projection plane rising from the outer edge of the planarsurface projection region in the Y direction in a parabolic shape. Inthis case, the stereoscopic projection plane rises in the Y directionfrom the front of the target parking region and the taken image of thetarget parking region is projected to the three-dimensional projectionplane. As illustrated in FIG. 8, the target parking region TR (e.g., aparking frame W included in the target parking region TR) included in acomposite image G displayed on the display 8 is, thus, distorted,thereby making it impossible to display the composite image G in whichthe target parking region TR of the vehicle 1 is visually recognizednaturally.

As illustrated in FIG. 11, the composite image generation unit 403projects the taken image to the stereoscopic projection plane includinga circular planar surface projection region having a radius of 30000 mmwith the position of the vehicle 1 as a center and a stereoscopicprojection plane rising from the outer edge of the planar surfaceprojection region in the Y direction in a parabolic shape. In this case,the target parking region is projected to the flat projection plane. Thethree-dimensional obstacle such as another vehicle present far from thetarget parking region when viewed from the position of the vehicle 1 isalso projected to the flat projection plane. As a result, as illustratedin FIG. 10, the image of the three-dimensional obstacle present far fromthe target parking region is not included in the composite image G whilethe target parking region TR (e.g., the parking frame W) included in thecomposite image G displayed on the display 8 is not distorted, therebymaking it impossible to display the composite image G that makes itpossible to identify the target parking region TR and the surroundingsof the vehicle 1.

As illustrated in FIG. 13, the projection region change unit 406enlarges the planar surface projection region such that the end on thefar side of the target parking region when viewed from the position ofthe vehicle 1 is included in the planar surface projection region. Thecomposite image generation unit 403 projects the taken image to thestereoscopic projection plane including a circular planar surfaceprojection region having a radius of 7000 mm with the position of thevehicle 1 as a center and a stereoscopic projection plane rising fromthe outer edge of the planar surface projection region in the Ydirection in a parabolic shape. In this case, the target parking regionis included in the planar surface projection region in the stereoscopicprojection plane and the three-dimensional obstacle present far from thetarget parking region when viewed from the position of the vehicle 1 isprojected to the stereoscopic projection region. As a result, asillustrated in FIG. 12, the target parking region TR included in thecomposite image G displayed on the display 8 is not distorted and thethree-dimensional obstacle present far from the target parking regioncan be included in the composite image G. The composite image G, thus,can be displayed in which the target parking region TR and thesurroundings of the vehicle 1 are visually recognized naturally.

The following describes other examples of the composite image displayedby the ECU 14 included in the vehicle 1 according to the embodiment withreference to FIGS. 14 to 19. FIGS. 14, 16, and 18 are diagramsillustrating the examples of the composite image displayed by the ECUincluded in the vehicle according to the first embodiment. FIGS. 15, 17,and 19 are exemplary cross-sectional views of the stereoscopicprojection plane to which the taken image is projected by the ECUincluded in the vehicle according to the first embodiment. In FIGS. 15,17, and 19, the abscissa axis represents the distance from the positionof the vehicle 1 to each point in the stereoscopic projection plane onthe XZ plane parallel to the X direction and the Z direction while thevertical axis represents the height (i.e., the coordinate in the Ydirection) of each point in the stereoscopic projection plane.

For example, as illustrated in FIG. 15, the composite image generationunit 403 projects the taken image to the stereoscopic projection planeincluding a circular planar surface projection region having a radius of5000 mm with the position of the vehicle 1 as a center and astereoscopic projection plane rising from the outer edge of the planarsurface projection region in the Y direction in a parabolic shape. Inthis case, the stereoscopic projection plane rises in the Y directionfrom the front of the target parking region and the target parkingregion is projected to the three-dimensional projection plane. Asillustrated in FIG. 14, the target parking region TR (e.g., the parkingframe W) included in the composite image G displayed on the display 8is, thus, distorted while another vehicle C present around the targetparking region TR is displayed naturally, thereby making it impossibleto display the composite image G in which the target parking region TRof the vehicle 1 is visually recognized naturally.

As illustrated in FIG. 17, the composite image generation unit 403projects the taken image to the stereoscopic projection plane includinga circular planar surface projection region having a radius of 30000 mmwith the position of the vehicle 1 as a center and a stereoscopicprojection plane rising from the outer edge of the planar surfaceprojection region in the Y direction in a parabolic shape. In this case,the target parking region is included in the planar surface projectionregion in the stereoscopic projection plane and the target parkingregion is projected to the flat projection plane. The other vehiclepresent around the target parking region and the three-dimensionalobstacle present far from the target parking region TR when viewed fromthe position of the vehicle 1 are also projected to the flat projectionplane. As illustrated in FIG. 16, the images of the other vehicle Cpresent around the target parking region TR and the three-dimensionalobstacle present far from the target parking region TR become elongatedimages while the target parking region TR (e.g., the parking frame W)included in the composite image G displayed on the display 8 is notdistorted. The composite image G, thus, cannot be displayed in which thetarget parking region TR and the surroundings of the vehicle 1 arevisually recognized naturally.

As illustrated in FIG. 19, the projection region change unit 406enlarges the planar surface projection region such that only the targetparking region TR in the stereoscopic projection region is included inthe planar surface projection region. The composite image generationunit 403 projects the taken image to the stereoscopic projection planein which the planar surface projection region is enlarged. In this case,the surroundings of the target parking region and the three-dimensionalobstacle present far from the target parking region when viewed from theposition of the vehicle 1 are projected to the stereoscopic projectionregion. As a result, as illustrated in FIG. 18, the target parkingregion TR (e.g., the parking frame W) included in the composite image Gdisplayed on the display 8 is not distorted and the surroundings (e.g.,the other vehicle C) of the target parking region TR and thethree-dimensional obstacle present far from the target parking region TRcan be included in the composite image G. The composite image G, thus,can be displayed in which the target parking region TR and thesurroundings of the vehicle 1 are visually recognized naturally.

As described above, the vehicle 1 according to the first embodimentallows the target parking region of the vehicle 1 and the movement pathof the vehicle 1 to be projected to the flat projection plane, therebymaking it possible to prevent the target parking region (e.g., theparking frame) of the vehicle 1 from being distorted in the stereoscopicprojection plane or not being projected to the stereoscopic projectionplane. The vehicle 1 according to the first embodiment, thus, candisplay the composite image in which the target parking region of thevehicle 1 is visually recognized naturally.

Second Embodiment

A second embodiment is an example in which the taken image is projectedto the stereoscopic projection plane using a 3D compatible map thatassociates the coordinates in the stereoscopic projection plane with thetexture coordinates of the taken image projected to the coordinates inthe stereoscopic projection plane and a 2D map that associates thecoordinates in the planar surface projection plane composed of only asingle projection plane parallel to the vehicle with the texturecoordinates of the taken image projected to the coordinates in theplanar surface projection plane. In the following description,description of the same structure as the first embodiment is omitted.

FIG. 20 is a diagram for explaining exemplary projection processing ofthe taken image by the ECU included in the vehicle according to thesecond embodiment. In the embodiment, the ROM 14 b is the storage unitthat stores therein a 3D map M1 and a 2D map M2. The 3D map M1 is a mapthat associates the coordinates in the stereoscopic projection planewith the texture coordinates of the taken image projected to thecoordinates in the stereoscopic projection plane. The 2D map M2 is a mapthat associates the coordinates in the planar surface projection planecomposed of only the single projection plane parallel to the vehicle 1with the texture coordinates of the taken image projected to thecoordinates in the planar surface projection plane.

In the embodiment, when the target parking region is decided (i.e.,parking support processing of the vehicle 1 is performed), theprojection region change unit 406 overwrites a map that associates thecoordinates of the movement path in the stereoscopic projection regionwith the texture coordinates of the taken image projected to thecoordinates of the movement path, the map being included in the 3D mapM1, on the 2D map M2 that associates the coordinates of the movementpath in the flat surface projection plane with the texture coordinatesof the taken image projected to the coordinates of the movement path.The projection region change unit 406 transmits the 3D map M1overwritten on the 2D map M2 to the composite image generation unit 403.In the embodiment, when the target parking region is not decided (i.e.,the parking support processing of the vehicle 1 is not performed), theprojection region change unit 406 transmits the 3D map M1 stored in theROM 14 b to the composite image generation unit 403 without overwritingthe 3D map M1 on the 2D map M2. This makes it unnecessary to calculatethe coordinates in the stereoscopic projection plane and the texturecoordinates of the taken image projected to the coordinates in thestereoscopic projection plane every time the composite image isdisplayed. As a result, the display processing of the composite imagecan be accelerated. Even when the movement path in the stereoscopicprojection plane is changed, the 3D map M1 is overwritten on the 2D mapM2 and thereafter the 3D map M1 is transmitted to the composite imagegeneration unit 403, thereby making it unnecessary to recalculate thecoordinates in the stereoscopic projection plane and the texturecoordinates of the taken image projected to the coordinates in thestereoscopic projection plane. As a result, the display processing ofthe composite image can be more accelerated.

In the embodiment, the composite image generation unit 403 projects thetaken image acquired by the image acquisition unit 402 to thestereoscopic projection plane in accordance with the 3D map M1 receivedfrom the projection region change unit 406. Specifically, the compositeimage generation unit 403 identifies, for each pixel included in thetaken image, the coordinates in the stereoscopic projection planeassociated with the texture coordinates of the pixel in the 3D map M1.The composite image generation unit 403 projects the taken image to thestereoscopic projection plane by projecting the pixel to the position atthe identified coordinates in the stereoscopic projection plane.

As described above, the vehicle 1 according to the second embodimentmakes it unnecessary to calculate the coordinates in the stereoscopicprojection plane and the texture coordinates of the taken imageprojected to the coordinates in the stereoscopic projection plane everytime the composite image is displayed. The vehicle 1 according to thesecond embodiment, thus, can accelerate the display processing of thecomposite image.

1. An image processing device, comprising: a processor configured to:acquire a taken image obtained by imaging surroundings of a vehicle byan imaging unit mounted on the vehicle; project the taken image to astereoscopic projection plane including the vehicle and a firstprojection region rising above the vehicle; produce a composite image ofthe stereoscopic projection plane when viewed from a virtual viewpoint;cause a display to display the composite image; determine a targetparking region in which the vehicle is parked in the stereoscopicprojection plane; and shift at least a part of a movement path of thevehicle from a position of the vehicle to the target parking region andthe target parking region of the vehicle to a second projection regionhorizontal to the vehicle.
 2. The image processing device according toclaim 1, wherein the first projection region is positioned around thesecond projection region, the second projection region includes thevehicle, and the processor enlarges the second projection region as awhole such that an end on a far side of the target parking region whenviewed from the position of the vehicle is included in the secondprojection region.
 3. The image processing device according to claim 1,wherein the processor partially enlarges the second projection regionsuch that only at least a part of the target parking region and themovement path in the first projection region is included in the secondprojection region.
 4. The image processing device according to claim 1,wherein the first projection region is positioned around the secondprojection region, the second projection region includes the vehicle,and the processor enlarges the second projection region such that an endon a far side of the target parking region when viewed from the positionof the vehicle is included in the second projection region when anobstacle is absent around the target parking region, and partiallyenlarges the second projection region such that only a part of thetarget parking region and the movement path in the first projectionregion is included in the second projection region when an obstacle ispresent around the target parking region.
 5. The image processing deviceaccording to claim 1, further comprising: a storage that stores thereina first map associating coordinates in the stereoscopic projection planewith texture coordinates of the taken image projected to the coordinatesand a second map associating coordinates in a planar surface projectionplane, the planar surface projection plane being composed of a singleprojection plane horizontal to the vehicle, with the texture coordinatesprojected to the coordinates, wherein the processor, when the targetparking region is decided, overwrites a map associating coordinates ofthe movement path in the second projection region with the texturecoordinates of the taken image projected to the coordinates, the mapbeing included in the first map, on the second map associatingcoordinates of the movement path in the planar surface projection planewith the texture coordinates of the taken image projected to thecoordinates, and projects the taken image to the stereoscopic projectionplane in accordance with the first map.